Enhancing the Transformer Decoder with Transition-based Syntax. (arXiv:2101.12640v3 [cs.CL] UPDATED)
Notwithstanding recent advances, syntactic generalization remains a challenge
for text decoders. While some studies showed gains from incorporating
source-side symbolic syntactic and semantic structure into text generation
Transformers, very little work addressed the decoding of such structure. We
propose a general approach for tree decoding using a transition-based approach.
Examining the challenging test case of incorporating Universal Dependencies
syntax into machine translation, we present substantial improvements on test
sets that focus on syntactic generalization, while presenting improved or
comparable performance on standard MT benchmarks. Further qualitative analysis
addresses cases where syntactic generalization in the vanilla Transformer
decoder is inadequate and demonstrates the advantages afforded by integrating
syntactic information.
( 2
min )